Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Stochastic neural networks (SNNs) are random functions whose predictions are gained by averaging over multiple realizations.

Abstract--Domain Incremental Learning (DIL) is a continual learning sub-branch that aims to address never-ending arrivals of new domains without catastrophic forgetting problems. Despite the advent of parameter-efficient fine-tuning (PEFT) approaches, existing works create task-specific LoRAs overlooking shared knowledge across tasks. Inaccurate selection of task-specific LORAs during inference results in significant drops in accuracy, while existing works rely on linear or prototype-based classifiers, which have suboptimal generalization powers. Our paper proposes continual knowledge consolidation low rank adaptation (CONEC-LoRA) addressing the DIL problems. CONEC-LoRA is developed from consolidations between task-shared LORA to extract common knowledge and task-specific LORA to embrace domain-specific knowledge. Unlike existing approaches, CONEC-LoRA integrates the concept of a stochastic classifier whose parameters are sampled from a distribution, thus enhancing the likelihood of correct classifications. Last but not least, an auxiliary network is deployed to optimally predict the task-specific LoRAs for inferences and implements the concept of a different-depth network structure in which every layer is connected with a local classifier to take advantage of intermediate representations. This module integrates the ball-generator loss and transformation module to address the synthetic sample bias problem. Our rigorous experiments demonstrate the advantage of CONEC-LoRA over prior arts in 4 popular benchmark problems with over 5% margins. ONTINUAL learning (CL) constitutes a research area of growing interests where the main goal is to develop a learning agent that can accumulate knowledge overtime [1], [2], [3], [4].

We thank the reviewers for the detailed feedback. R1 and R2 ask for a comparison to a prior method. R3's main comments are about the theoretical results. We've addressed all of this below. We are thus able to handle a more general set of constraints, including those for ranking fairness.

Stochastic classifiers arise in a number of machine learning problems, and have become especially prominent of late, as they often result from constrained optimization problems, e.g. for fairness, churn, or custom losses.

We present a general framework for solving a large class of learning problems with non-linear functions of classification rates.

Stochastic neural networks (SNNs) are random functions whose predictions are gained by averaging over multiple realizations.